Imaging based stabilization

ABSTRACT

An image-based sensor system for a mobile unit makes use of light emitters and imagers to acquire illumination patterns of emitted light impinging on the floor and/or walls surrounding the unit. The illumination pattern is used to estimate location and/or orientation of the unit. These estimates are used for one or more functions of stabilization, calibration, localization, and mapping of or with respect to the unit.

BACKGROUND

This invention relates to stabilization using an imaging input.

Stabilization, calibration, localization, and mapping functions may bedifficult in an interior environment, for example, because of the lackof reliable signals from a positioning system (e.g, a Global PositioningSystem), and because many inertial sensors suffer from characteristicssuch as signal drift.

One application is for use on an aerial vehicle that is designed to flyin interior environments. An example of such a vehicle is found in U.S.Pat. No. 7,631,834, issued on Dec. 15, 2009. There is a need to providea sensor system for such an aerial vehicle that is small, lightweight,and cost effective.

SUMMARY

In one aspect, in general, a mobile part of an apparatus has a firstlight emitter for forming a first light emission. The first lightemission includes emitted rays on a cone or cylinder that is symmetricalabout a central axis of the mobile part. The mobile part includes afirst imager for acquiring images from a point of view along the centralaxis showing an illumination pattern at intersections of the emittedrays and a first surface upon which the first light emission impinges. Afirst estimator of the apparatus is configured for processing theacquired images and using the acquired images to determine at least oneof (a) an orientation of the central axis of the apparatus with respectto the first surface, and (b) a distance from the surface along thecentral axis.

Aspects may include one or more of the following features.

The mobile part comprises an aerial vehicle, and the apparatus furthercomprises a module configured to be responsive to the determinedorientation and/or distance to perform at least one function from agroup of functions consisting of stabilization, calibration,localization, and mapping.

The module responsive to the determined orientation and/or distancecomprises a control system configured to generate control signals forflight controls of the aerial vehicle.

The control system is configured to be responsive to the determinedorientation to stabilize an orientation of the vehicle.

The control system is configured to be responsive to the determineddistance to stabilize an altitude of the vehicle above the surface.

The first light emitter comprises multiple optical elements disposed ata fixed radius perpendicular from a point on the central axis.

The multiple optical elements include multiple laser light sources.

The multiple optical elements include multiple reflectors, and the firstlight emitter further includes a laser light source configured toreflect rays off the reflectors to form the first light emission.

The estimator is configured to determine the orientation according to aneccentricity of the illumination pattern.

The estimator is configured to determine the distance according to asize of the illumination pattern.

The first light emitter includes a reflector configured to rotate aboutthe central axis, and a laser light source configured to reflect a rayoff the rotating mirror to form the rays on the cone that is symmetricalabout the central axis.

The mobile part comprises an aerial vehicle having a propeller assemblyconfigured to rotate about the central axis to provide lift for theaerial vehicle, and the reflector is coupled to the propeller assemblyto rotate with the rotation of the propeller assembly.

The mobile part further includes a second light emitter for forming asecond light emission comprising a plurality of emitted rays from thecentral axis of the mobile part. The mobile part also includes a secondimager for acquiring images showing an illumination pattern atintersections of the emitted rays of the second light emission and oneor more second surfaces upon which the second light emission impinges.The apparatus further includes a second estimator for processing theacquired images for the second imager and using the acquired images todetermine a location of the mobile part relative to the one or moresecond surfaces.

The rays of the second emission are on a plane that is perpendicular tothe central axis of the mobile part.

The second imager acquires images from one or more points of view thatare displaced in the direction of the central axis from the plane of therays of the second light emission.

The estimator is configured to determine the location relative to thesurface according to a displacement of the displacement in the imagesacquired by the second imager of the illumination pattern in a directioncorresponding to the central axis.

The second imager comprises a plurality of imagers disposed about theperiphery of the mobile part, and wherein the second imager isconfigured to combine the images of the plurality of imagers to form animaging including all directions extending from the central axis.

The second estimator is configured to be responsive to the location toperform at least one function from a group of functions consisting ofstabilization, calibration, localization, and mapping.

The first estimator is implemented in the mobile part of the apparatus.

The mobile part comprises an aerial vehicle that is tethered to acontrol station, and wherein the first estimator is implemented at leastin part at the control station.

In another aspect, in general, a method for calibrating a image-guidedapparatus includes placing the apparatus in a calibration unit. Thecalibration unit has a substantially cylindrical inner surface and theapparatus is placed in the calibration unit to align a central axis ofthe apparatus with the central axis of the cylinder. A light emission isemitted from a light emitter affixed to the apparatus. The lightemission comprises rays in a symmetrical pattern about the central axis.The rays impinge on the inner surface of the cylinder. Images areacquired from an imager affixed to the apparatus. The images show anillumination pattern at intersections of the emitted rays and the innersurface of the cylinder. An estimator of the apparatus is calibratedaccording to known dimensions of the calibration unit and theillumination pattern.

Aspects may have one or more of the following advantages.

Relatively inexpensive commercially available cameras can be used toacquire images of the laser illumination. The resolution of such camerasis sufficient such that the pattern of the illumination can be used instabilization/control, mapping, and localization algorithms. The framerate of available camera units is sufficient to provide measurementupdates for such real-time algorithms.

An easily implemented calibration procedure, for instance using acalibrated cylindrical unit, allows the light emitters and imagers to beaffixed to the mobile part without requiring precise placement.

An aerial vehicle can be stabilized using the approach, for example,incorporating the determined location, altitude, or orientation in acontrol loop.

Other features and advantages of the invention are apparent from thefollowing description, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a side cross-sectional view of an aerial vehicle in a indoorenvironment;

FIG. 2 is a top view corresponding to FIG. 1;

FIG. 3 is a schematic diagram of a side image sensor output;

FIG. 4 is a side cross-sectional view of the aerial vehicle is a secondpose;

FIG. 5 is a schematic diagram of a bottom image sensor output; and

FIG. 6 is a diagram that illustrates sideways imaging.

FIG. 7 is a diagram showing a laser generator means or projectingdownward circles or horizontal lines from an aerial vehicle.

DESCRIPTION

Referring to FIG. 1 number of approaches to stabilization, calibration,and/or localization are described in the context of an aerial vehicle100. In some embodiments, the vehicle 100 is similar to vehicles thatare described in U.S. Pat. No. 7,631,834, “Aerial Robot with DispensableConductive Filament,” issued on Dec. 15, 2009. In general, as describedmore fully later in this document, the vehicle includes a controller tomaintain stability and to move in a controlled manner (e.g., accordingto a remote control or autonomously) based on sensor inputs that includeinertial sensors that measure linear and rotational acceleration. Thisdocument describes use of optical sensing of the environment to provideadditional inputs to the controller and/or for other uses includingcalibration of sensors, mapping of the environment and localization ofthe vehicle in the environment.

Generally, the vehicle 100 has body 110 with a generally cylindricaldesign that forms a duct through which fans 130 drive air through theduct to lift the vehicle. The vehicle 100 includes a number of controlsurfaces (not shown) that are adjusted by the controller to maintainstable controlled flight. The vehicle 100 is shown schematically in FIG.1 with a central member 120 that extends along the axis of the vehicle.

The vehicle 100 shown in FIG. 1 includes a number of light emitters.Other embodiments may include fewer or more than those shown. A firstlight emitter 155 is mounted on the central axis of the vehicle andproduces planar rays 157 that emit perpendicularly to the central axisin all directions. In some examples, the light emitter includes a lasersource (e.g., 1-300 mW at 980 nm wavelength) coupled to a 360 degreeline generator. Typically, the rays 157 impinge on a wall 185illuminating a line 158 (shown in cross-section as a point in FIG. 1).

The vehicle includes an imager that is made up of a number of cameras150 disposed around the peripheral surface of the vehicle, for example,with eight cameras being arranged around the vehicle so that inaggregate they provide a 360 degree acquisition angle. Two of thecameras are shown in cross-section in FIG. 1. As an illustration, thevehicle 100 is oriented with its central axis parallel to a wall 185,and the wall 185 is a distance x from the camera 150. An illuminatedpoint 158 is a height y above the height of the camera 150, and a lightray 152 reflects from the point 158 to the camera 150. Another wall 188is a distance x′ from the camera with a light ray 153 reflecting to thecamera.

The vehicle also includes a second light emitter 165 that producesdownward rays 167 parallel to the central axis at a radius r from thecentral axis. In some examples, the rays 167 are continuous around theentire circumference, thereby forming a cylindrical ray that impinges ona floor 180. In some examples, the light emitter 165 is formed with acentral laser source and reflectors (e.g., mirrors, prisms) axially oralong the circumference to direct the rays downward. In someembodiments, a number of discrete rays 167 are generated by separatelaser sources. In the cross section shown in FIG. 1, the light rays 167intercept the floor 180 at points 168.

The vehicle includes a camera 160 that points downward from a point ofview on the central axis. Reflected light rays 162 pass from the points168 on the floor to the camera 160. In the illustration, the camera 160is at a height h above the floor.

Referring to FIG. 2, a top view of the vehicle 100 shown in FIG. 1 showsthe cylindrical body 110 of the vehicle, and the arrangement of eightcameras 150 around the periphery of the body. The walls 185, 188 form acorner such that the wall 185 is a distance x from the camera 150 andthe wall 188 is a distance x′ from the camera. Representative rays 157(of the full planar ray that emits in all directions) are shown in thefigure.

Referring to FIG. 3, a portion of a panoramic image 310 (e.g., less than90 degrees for the panorama) formed from the images from the set ofcameras 150 shows the image of the ray 157 as an interrupted line, witha line 322 corresponding to the reflection from the wall 185 and a line324 corresponding to the reflection from the wall 188. Referring to thegeometry shown in FIG. 1, the height of the line 322 is proportional to

$\frac{1}{x}$and the height of the line 324 is proportional to

$\frac{1}{x^{\prime}}.$To the extent that the height of the refection can be located in thepanoramic image, the distance from the central axes to the wall on theplane of the light ray 157 can be determined from the panoramic image asa function of the angle φ around the central axis. A procedure forcombining the separate images from each of the cameras 150 to form apanoramic image is discussed further later in this document.

Referring again to FIG. 1, when the vehicle's central axis isperpendicular to the floor 180, the downward rays 167 intersect thefloor to form a circle. The radius of the circle remains constant butthe image of the circle is proportional to

$\frac{1}{h}$such that as the vehicle rises, the image of the circle appears to growsmaller. Therefore, the radius of the circle provides a measure ofaltitude of the vehicle above the floor.

Referring to FIG. 4, when the vehicle 100 is tilted, the downward raysdo not intercept the floor in a circle. Referring to FIG. 5, when in avertical pose at a height h, the image of the circle 520 have a radius

$\frac{1}{h}.$When the vehicle is tilted the intersection of the light rays 167 formsan ellipse on the floor. With heights are h1 and h2 at the twointersecting points in the cross section shown in FIG. 4, the image ofthe ellipse 530 has a major axis extending

$\frac{1}{h\; 1}$in one direction and

$\frac{1}{h\; 2}$in the other direction from the central point of the image 510.

Generally, in some examples, the images of the cameras are used toprovide estimates of range (x) as a function of direction relative tothe vehicle (φ) and altitude (h) as a function of direction (φ) alongthe circumference of the body. In some examples, the images areprocessed to derive features, such as distance to discontinuities ordiscontinuities in direction of the horizontal lines that correspond tocorners of walls, or average altitude assuming a planar floor, directionand magnitude of tilt of the vehicle.

In some examples, the cameras are lightweight CMOS imagers, such as orsimilar to those used in digital cameras or cameras in cellulartelephones. In some examples, the imagers are direct digital buscompatible and are physically smaller than other alternatives, such asCCD based imagers. A specific example used in some embodiments is anOmnivision OV10620, which has over 100 dB of luminance dynamic range.Peak quantum efficiency exceeds 35% in the 980 nm IR spectrum, whichcoupled with the imagers low light performance, allows the use of narrowbandwidth optical filters (≦7 nm) to perform well in direct sunlightapplications.

The Omnivision 10620 imager has a pixel resolution of H×V=768×506. Thedistance sensing resolution of the system can be understood byconsidering the effect of a one-pixel vertical displacement in an image.Referring to FIG. 6, a representative camera image 610 is shown suchthat the image of the reflected point appears at a top pixel location ofthe image when the distance to the wall is x_(min). For the sake ofillustration, if we assume that the distance from the optical centerlineto the ray plane (y) is 0.4 m, and that the reflected point isapproximately θ_(min)=22° above the optical centerline when the wall isat the distance x_(min) then x_(min)≈0.4 m/tan 22°≈1.0 m. At thisminimum distance, one pixel difference vertically corresponds toapproximately 4 mm. This resolution goes down approximatelyproportionally to x² so that at a distance of x=10 m, in thisillustrative example, one pixel vertical difference corresponds toapproximately 40 cm.

Note also that in an example in which eight cameras are disposed aboutthe circumference of the vehicle, each camera is responsible for atleast 45 degrees of view, and the cameras have overlapping fields toprovide continuity and calibration as described below. For example, inthe illustrative example above in which V/2=253 pixels correspond to 22degrees, then H=768 pixels corresponds to about 63 degrees, providingabout 9 degrees of overlap on each edge.

Note that because of the geometry of the camera sensors, for reflectedpoints in images displaced horizontally by an angle φ from the opticalcenterline, the pixel height of the image of the reflection is notproportional to

$\frac{1}{x}$but rather

$\frac{\cos\;\phi}{x}.$That is, a circular wall a constant distance from the vehicle does notresult in a straight horizontal line on the sensor image.

In some embodiments, an image processing step is performed to transformthe individual sensor images to form a panorama image that compensatesfor the distortions inherent in geometry. In some embodiments, acalibration step is performed such that each pixel (h, v) maps to aparticular angle and range (φ, x).

In one example of calibration, the vehicle is place such that itscentral axis is aligned in a cylinder that has a calibrated diameter(e.g., 2 m), and optionally includes calibrations for the angle φ aboutthe circle. During the calibration phase, images are acquired and the(h, v)→(φ, x) map is computed for each sensor. This calibration canaccount for various unknown parameters for each of the cameras, such asthe deviation of the optical axis and skew around that axis as comparedto a precise positioning of the cameras. This can relieve the precisionwith which the cameras are mounted to the vehicle body. In some examplesof calibration, the overlapping portions of the camera images are usedin the calibration process.

A similar calibration process is used for the downward facing sensor, toestablish the image pixel (h₀, v₀) corresponding to the image of a pointon the central axis, and an orientation of the sensor around the centralaxis (φ₀), and an image radius corresponding to a standard height (e.g.,1 m).

In some embodiments, the sensor images are used as inputs to acontroller that provides control signals to the control surfaces of thevehicle. Periodically, for example, 24-60 times per second, the imagesare acquired concurrently by all the cameras and the images aretransferred over a bus (e.g., by DMA access) to a processor (e.g., anNXP3250 multimedia processor or a DSP Digital Signal Processor) on thevehicle that processes the images.

The image processing includes detecting the locations of the reflectionpoints in the images. In some examples, the processing also includesidentifying features of the reflections, for example, discontinuities ofvarious types that may correspond to corners in walls or floors. Thelocations for the reflection points are then provided as inputs to acontrol algorithm, which in some examples implements an extended KalmanFilter. Generally, the Kalman Filter makes use to a current estimate ofthe location and orientation of the vehicle to form differences betweenthe inputs (such as (φ, x) pairs from the sideways looking sensors) andthe predicted values based on the estimate.

In some embodiments, the image sensor signals are used in calibration ofinertial sensors, for example, to compensate for drift in the inertialsensors on an ongoing basis.

In some examples, a Simultaneous Localization and Mapping (SLAM)approach is used in which the vehicle builds up a map of the environmentwhile at the same time localizing its location in the map.

FIG. 7 shows is a means for utilizing the existing rotational elementsof an aerial vehicle to create a lightweight and low cost laser patterngenerator by affixing a reflective mirror 760 to a propeller rotationalelement (e.g., below the propellers as illustrated, or alternativelyabove the propellers) so that the mirror rotates along with the mirror,for example, at the same rotation speed. A collimated light (orholographically generated structured light pattern) is directed from alight source 160 at the mirror 760. By selecting the mirror mountingangle ψ, a range of downward projecting cones intersecting the ground isachieved. By making ψ nearly 45 degrees, a horizontal planar rayperpendicular to the rotation axis is likewise generated. The laserlight source 160 can be mounted either on axis as shown, oralternatively substantially off axis without loss of effectiveness. Aslong as the field of view (FOV) optical angle of downward imager 165 isdifferent from the projected angle cone of light ρ there exists a uniquesolution to calculate both vehicle height and vehicle attitude asbefore.

In some embodiments, other patterns for illumination are used. Forinstance, the sideways and/or downward rays form cones rather thanplanes and cylinders, respectively. In some embodiments, the sidewaysprojecting rays form two or more lines on the walls, for example, byprojecting two planes of rays at different vertical displacements, or byprojecting rays on one or more cones that are symmetrical about thecentral axis of the vehicle. In other embodiments, other forms of lightpatterns are used, for example, by using holographic projectionstechniques.

The techniques described above are not necessary limited for use on anaerial vehicle. For example, the cameras described above may be affixedto a shaft that is carried in a generally vertical orientation, affixedto a helmet, ground robot, etc. The processing of the sensor inputs canthen be used for a SLAM algorithm that builds a map of an interior space(e.g., the interior of a building), and maintains an estimate of alocation within the building.

In some examples, the processing steps described above are implementedin software on a computer readable medium (e.g., disk, non-volatilesemiconductor memory etc.) that includes instructions that controlexecution of a computer processor, for instance a general purposeprocessor, microprocessor, digital signal processor, and/or multimediaprocessor. In some examples, some steps are performed using hardware,for instance, using application specific integrated circuits (ASICs).

In some examples, the processing is distributed between multiplelocations, for example, being distributed between the aerial vehicle anda control location that is in communication with the vehicle, forinstance, over a micro-filament communication link. For example, imageprocessing may be performed on the vehicle, with distance estimatesbeing sent to the control location for ongoing calibration ofvehicle-based sensors.

It is to be understood that the foregoing description is intended toillustrate and not to limit the scope of the invention, which is definedby the scope of the appended claims. Other embodiments are within thescope of the following claims.

What is claimed is:
 1. An apparatus comprising mobile part, the mobilepart comprising: a first light emitter for forming a first lightemission comprising a plurality of emitted rays, each ray being emittedin a direction along one of a plurality of lines parallel to a centralaxis of the mobile part or divergent from a common point on said axis,the plurality of rays exhibiting a symmetry about said axis; and a firstimager for acquiring images from a point of view along the central axisshowing an illumination pattern at intersections of the emitted rays anda first surface upon which the first light emission impinges; whereinthe apparatus further includes a first estimator for processing theacquired images and using the acquired images to determine at least oneof (a) an orientation of the central axis of the apparatus with respectto the first surface, and (b) a distance from the surface along thecentral axis.
 2. The apparatus of claim 1 wherein the mobile partcomprises an aerial vehicle, and wherein the apparatus further comprisesa module configured to be responsive to the determined orientationand/or distance to perform at least one function from a group offunctions consisting of stabilization, calibration, localization, andmapping.
 3. The apparatus of claim 2 wherein the module responsive tothe determined orientation and/or distance comprises a control systemconfigured to generate control signals for flight controls of the aerialvehicle.
 4. The apparatus of claim 3 wherein the control system isconfigured to be responsive to the determined orientation to stabilizean orientation of the vehicle.
 5. The apparatus of claim 3 wherein thecontrol system is configured to be responsive to the determined distanceto stabilize an altitude of the vehicle above the surface.
 6. Theapparatus of claim 1 wherein the first light emitter comprises aplurality of optical elements disposed at a fixed radius perpendicularfrom a point on the central axis.
 7. The apparatus of claim 6 whereinthe plurality of optical elements comprise a plurality of laser lightsources.
 8. The apparatus of claim 6 wherein the plurality of opticalelements comprise a plurality reflectors, and the first light emitterfurther comprises a laser light source configured to reflect rays offthe plurality of reflectors to form the first light emission.
 9. Theapparatus of claim 1 wherein the estimator is configured to determinethe orientation according to an eccentricity of the illuminationpattern.
 10. The apparatus of claim 1 wherein the plurality of lines liein a cone or cylinder.
 11. The apparatus of claim 10 wherein the firstlight emitter comprises a reflector configured to rotate about thecentral axis, and a laser light source configured to reflect a ray offthe rotating minor to form the rays in the cone that is symmetricalabout the central axis.
 12. The apparatus of claim 11 wherein the mobilepart comprises an aerial vehicle having a propeller assembly configuredto rotate about the central axis to provide lift for the aerial vehicle,and wherein the reflector is coupled to the propeller assembly to rotatewith the rotation of the propeller assembly.
 13. The apparatus of claim1 wherein the mobile part further comprises: a second light emitter forforming a second light emission comprising a plurality of emitted raysfrom the central axis of the mobile part; and a second imager foracquiring images showing an illumination pattern at intersections of theemitted rays of the second light emission and one or more secondsurfaces upon which the second light emission impinges; wherein theapparatus comprises a second estimator for processing the acquiredimages for the second imager and using the acquired images to determinea location of the mobile part relative to the one or more secondsurfaces.
 14. The apparatus of claim 13 wherein the rays of the secondemission are on a plane that is perpendicular to the central axis of themobile part.
 15. The apparatus of claim 14 wherein the second imageracquires images from one or more points of view that are displaced inthe direction of the central axis from the plane of the rays of thesecond light emission.
 16. The apparatus of claim 15 wherein theestimator is configured to determine the location relative to thesurface according to a displacement of the displacement in the imagesacquired by the second imager of the illumination pattern in a directioncorresponding to the central axis.
 17. The apparatus of claim 13 whereinthe second imager comprises a plurality of imagers disposed about theperiphery of the mobile part, and wherein the second imager isconfigured to combine the images of the plurality of imagers to form animaging including all directions extending from the central axis. 18.The apparatus of claim 13 wherein the second estimator is configured tobe responsive to the location to perform at least one function from agroup of functions consisting of stabilization, calibration,localization, and mapping.
 19. The apparatus of claim 1 wherein thefirst estimator is implemented in the mobile part of the apparatus. 20.The apparatus of claim 1 wherein the mobile part comprises an aerialvehicle that is tethered to a control station, and wherein the firstestimator is implemented at least in part at the control station. 21.The apparatus of claim 1 wherein the estimator is configured todetermine the distance according to a size of the illumination pattern.22. A method for calibrating an image-guided apparatus, comprising:placing the apparatus in a calibration unit, the calibration unit havinga substantially cylindrical inner surface, the apparatus being placed toalign a central axis of the apparatus with the central axis of thecylinder; emitting a light emission from a light emitter affixed to theapparatus, the light emission comprising rays in a symmetrical patternabout the central axis, the rays impinging on the inner surface of thecylinder; acquiring images from an imager affixed to the apparatus, theimages showing an illumination pattern at intersections of the emittedrays and the inner surface of the cylinder; and calibrating an estimatorof the apparatus according to known dimensions of the calibration unitand the illumination pattern.